DOCUMENT RESUME 



CS 510 654 

St. John, Edward P.; Manset, Genevieve; Chung, Choong-Geun; 
Worthington, Kimberly 

Assessing the Rationales for Educational Reforms: A Test of 
the Professional Development, Comprehensive Reform, and 
Direct Instruction Hypotheses. Policy Research Report. 
Indiana Univ. , Bloomington. Education Policy Center. 

Indiana State Dept, of Education, Indianapolis. 

RR- 01 - 03 
2001.-09-00 
4 Op . 

Indiana Education Policy Center, Smith Center for Research 
in Education, Suite 170, 2805 E. 10th St., Indiana 
University, Bloomington, IN 47408-2698. Tel: 812-855-1240; 
Web site: http://www.indiana.edu/~iepc/. 

Information Analyses (070) 

MF01/PC02 Plus Postage. 

*Early Intervention; ^Educational Change; Primary Education; 
*Prof essional Development; ^Program Effectiveness; ^Reading 
Instruction; ^Remedial Reading 

^Direct Instruction; Even Start; Four Blocks Approach; 
Reading Recovery Projects 

Educational reforms are advocated based on rationales that 
emerge from the research literature. However, evaluation studies seldom 
examine whether the rationales used to argue for a reform actually hold up 
when empirical evidence is examined after the reform has been implemented. 
This paper examines survey data from 3 years of analyses of early reading 
interventions to examine 3 of the rationales that were used to argue for the 
program. First, there was evidence to support the argument that teachers need 
time to collaborate about improving educational outcomes. This study found 
that 2 years of funding for early reading reforms (Reading Recovery, Success 
for All, Literacy Collaborative, full day kindergarten, First Steps, Even 
Start, Accelerated Schools Project, and Four Blocks) provided a margin of 
difference for collaborative efforts among teachers to promote reading 
related outcomes. Second, the argument that comprehensive reform strategies 
promote gains in student outcomes was supported, but not all reform models 
had their intended effects. Finally, there was no evidence from this study 
that the direct/explicit approach to reading instruction improved student 
outcomes, although this claim merits more systematic study in the future. 
(Contains 56 references, 7 notes, and 6 tables of data.) (Author/RS) 



ED 458 641 

AUTHOR 

TITLE 



INSTITUTION 
SPONS AGENCY 
REPORT NO 
PUB DATE 
NOTE 

AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



o 

ERJC 



■.'I.T.LT.'.TllIJ 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



0 654 



-ii — - rr - ~~ — ■ ■ «- ■ - --r. - i v- ■ » . , ft 

n 

Assessing the Rationales 
< for Educational Reforms: 

A Test of the Professional Development, 
Comprehensive Reform, 
and Direct Instruction Hypotheses 






PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



M. 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



ERLC 



° Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




Web site: http://www.indiana.edu/Mepc 



2 



Assessing the Rationales 
for Educational Reforms: 

A Test of the Professional Development, 
Comprehensive Reform, 
and Direct Instruction Hypotheses 



By 

Edward P. St. John 
Genevieve Manset 
Choong-Geun Chung 
Kimberly Worthington 

Indiana Education Policy Center 
Indiana University 



September 200 1 
Policy Research Report #01-03 



Acknowledgments: The research reported herein was completed with the support of the Indiana Department 
of Education, State of Indiana. The interpretations in this paper are the authors and do not necessarily 
represent policies, positions, or views of the Indiana Department of Education any other agency of the State 
of Indiana. 



Assessing the Rationales for Educational Reforms: 

A Test of the Professional Development, Comprehensive 
Reform, and Direct Instruction Hypotheses 



Abstract 

Educational reforms are advocated based on rationales that emerge from the 
research literature. However, evaluation studies seldom examine whether the rationales 
used to argue for a reform actually hold up when empirical evidence is examined after the 
reform has been implemented. This paper examines survey data from three years of 
analyses of early reading interventions to examine three of the rationales that were used 
to argue for the program. First, there was evidence to support the argument that teachers 
need time to collaborate about improving educational outcomes. This study found that 
two years of funding for early reading reforms provided a margin of difference for 
collaborative efforts among teachers to promote reading related outcomes. Second, the 
argument that comprehensive reform strategies promote gains in student outcomes was 
also supported, but not all reform models had their intended effects. Finally, there was no 
evidence from this study that the direct/explicit approach to reading instruction improved 
student outcomes, although this claim merits more systematic study in the future. 



Introduction 



Educational reform in the United States is at an interesting, but perplexing, point 
in its evolution. The current wave of school reform is, in theory at least, research-based, 
which provides an opportunity for educational researchers, and perhaps even educators, 
to test new approaches. Recent developments in reading and comprehensive reform 
provide teachers — and their schools — with a range of choices about the types of reform 
strategies they may want to undertake. However, there is a long history of failure of 
education reforms, especially in urban schools (Miron & St. John, in press), which 
complicates efforts to develop policies and strategies to guide the choices educators 
might make about reform strategies. Different and potentially competing rationales have 
been used to argue to the new wave of reading and comprehensive reforms. This 
complicates efforts to evaluate the impact of education reforms. 

Three arguments about reform seem to have a substantial influence on the way 
recent reading and comprehensive reforms have been conceptualized within state and 
federal programs, implemented by states and schools, and evaluated. One of these 
paradigms is that professional development, especially providing opportunities for 
educators to collaborate in schools, can foster improvement in educational outcomes 
(Bull & Buechler, 1996; Education Commission of the State, 1997; Guskey & Sparks, 
1997). A second is that comprehensive models that engage teachers in cohesive reform 
strategies increase the chances more students will succeed (e.g., Clay, 1993; Taylor, 
Anderson, Au, & Jaffy, 2000). A third is that direct, explicit instruction in letter/sound 
relationships is necessary to foster reading by third grade, a strategy that is essential to 
move more children into the educational mainstream (Foorman, Fletcher, Francis, & 
Schatschneider, 2000; Snow, Griffith, & Bums, 1998). 

While these rationales are actually comprised of micro sets of competing reform 
arguments, each of the rationales is used by a general belief community. For example, 
while there may be substantial disagreements about philosophies and strategies among 
the advocates for comprehensive reforms (e.g.. Success for All and Accelerated Schools), 
they are in a coalition of advocates for comprehensive reform. Also, with reading reform, 
there are long-standing disputes about direct instruction, but many reforms aligned with a 







literature-rich philosophy (e.g., Reading Recovery) benefit from the new advocacy for 
direct instruction, which has influenced new state and federal investments in reading 
reform. Thus, all three rationales influence state and federal reform efforts. Indeed, all 
three rationales have been used in states to promote early reading reform. 

Once beliefs reach the point of having pervasive influence on the rationales used 
for education policy development, the underlying claims may move along unexamined by 
educators, evaluators, and policymakers. This paper treats these rationales as 
“hypotheses” and uses the analyses of surveys of early reading programs in Indiana’s 
elementary schools to test these hypotheses. We describe the three hypotheses, present 
the study approach used to collect and analyze results, discuss the statistical analyses, 
reconsider the hypotheses in relation to the empirical evidence, and summarize by 
considering the implications of the study for research and educational policy. 

Competing Reform Hypotheses 

While it is widely recognized that competing theories, or hypotheses, guide 
reform efforts, researchers seldom investigate competing rationales for reform. To 
achieve this goal, it is necessary to discern the assumptions made by the reform 
advocates, as well as to analyze data that permits an objective testing of these 
assumptions. This is a complicated task because most of the research on reading reforms 
holds implicit advocacy positions. Thus, not only is it difficult to conceptualize research 
in ways that make it possible to assess different assumptions about reform, but it can be 
even more difficult to find data that can be used to test these assumptions. 

Instead of testing the rationales for reforms, most researchers conduct 
confirmatory research related to a specific reform model. By testing a single hypothesis 
rather than examining empirical evidence in relation to competing hypotheses, 
researchers eventually play the role of providing confirmatory evidence for individual 
reformers. However, the claims made as part of the rationale for funding often go 
untested. To examine the effects of rationales, we must start by examining the political 
arguments that influence the evolution of a funding program. 

This study examines the impact of a state early reading reform program on three 
school outcomes (special education referral for grades K-3, retention rates for grades K-3, 
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and pass rates on the state 3 rd grade reading achievement test). Three different rationales 
for education reform influenced the evolution of this state grant program. This paper 
examines three years of surveys from schools funded in the Indiana Early Intervention 
Grant Program (EIGP), along with surveys from comparison schools. Below we outline 
the three arguments and consider how these arguments influenced the EIGP. 

Professional Development 

The professional development hypothesis essentially is that teachers need time for 
professional development that involves collaboration on strategies for improving 
educational outcomes. In the middle 1990s, the Indiana Department of Education (IDOE) 
initiated efforts to support professional development for teachers. Initially, they studied 
state policies on professional development, focusing on release time for teachers in 
various states (Bull, Buechler, Didley, & Krehbiel, 1994). The IDOE has continued to 
advocate for subsidizing release time for teachers’ professional development. As a second 
phase of their advocacy process, the IDOE commissioned a study of the research base for 
professional development. This study found that “successful” professional development 
is: 

• School-based, that is, focused on particular problems of each school and 
selected by the teachers and principal to help address those problems. 

• Followed up in teachers’ classrooms by such means as peer coaching or school- 
based research and evaluation teams. 

• Collaborative, so that groups of professionals at schools can work together to 
solve the school problems most critical to student learning. 

• Embedded in the daily lives of teachers, so that they undertake professional 
growth as a central responsibility of teaching. 

• Focused on student learning, so that teams of teachers at the school attend to the 
actual effects of professional development on the performance of the students 
for whom they are responsible. (Bull & Buechler, 1996, p. 5). 

These principles have been widely disseminated in Indiana and nationally. In 
Indiana, the IDOE strongly encourages schools to consider these principles whenever 
they submit proposals for categorical funds (St. John, Ward, & Laine, 1999). Therefore 



the professional development rationale was embedded in the program and was integral to 
the funding process in EIGP. To receive a grant, a school either received a subsidy for 
professional development in Reading Recovery, or needed to include a focus on 
professional development (consistent with the principles above) in their grant application. 

Unfortunately, few conceptual frameworks have been developed for the analysis 
of the effects of professional development on student outcomes. These conceptual models 
argue that professional development has a direct influence on professional practices and 
influences student outcomes as a result of changes in educational practice outcomes 
(Guskey & Sparks, 1997; St. John, Ward, & Laine, 1999). This conceptualization is 
consistent with the principles outlined above and thus had an influence on the conceptual 
model for this study. We included an explicit examination of professional development 
activities as part of our surveys. 

Comprehensive Reform 

A second reform hypothesis is that comprehensive and cohesive reform 
approaches are essential to improving schools’ outcomes. In the debates over reading 
intervention strategies, Barbara Taylor and her colleagues (Taylor, Anderson, Au, & 
Taffy, 2000) have argued this position. In recent years, the comprehensive reform 
argument has also had a substantial influence on the Comprehensive School 
Restructuring Demonstration and schoolwide Title I (Wong, in press). This reform 
argument had a substantial influence on EIGP in Indiana. In particular, advocates of the 
Reading Recovery program influenced the creation of EIGP in Indiana. Reading 
Recovery was made the first priority for funding. Over time, about half of the funds were 
used for Reading Recovery and about half for other programs. 

In addition to the EIGP funding, the schools in Indiana had access to other types 
of funding from other state and federal sources, so it was possible that comparison 
schools included in these surveys had similar reforms funded by alternative programs. 
Therefore it was necessary to review the features of diverse reform models before we 
could develop a survey that could be used to analyze the effects of the different types of 
reform models that schools might use. The reforms that are represented in the current 
study are described briefly below. 
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Reading Recovery: A pullout, one-on-one reading intervention for the lowest- 
achieving students in first grade (i.e., the lowest 20%). Reading Recovery (RR) is 
designed to bring those students up to grade level. The intervention helps children make 
the difficult transition from decoding to comprehension. The program has been widely 
studied by proponents and independent researchers (e.g., Center, Wheldall, Freeman, 
Outhred, & McNaught, 1995; Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994; Iverson & 
Tunmer, 1993; Rowe, 1997). 

Success for All: A comprehensive school restructuring process designed for 
schools with large populations at risk for learning failure. Success for All (SFA) balances 
a skills-oriented instructional approach with a heavy emphasis on collaboration and 
teamwork among educators. The program has a substantial research base (Dianda & 
Flaherty, 1995; Madden, Slavin, Karweit, Donlan, & Wasik, 1991; Madden, Slavin, 
Karweit, Limermon, & Donlan, 1989; Ross & Smith, 1994) mostly conducted by 
advocates for SFA. 

Literacy Collaborative: A schoolwide restructuring model that focuses on 
classroom-based instruction, depending on Reading Recovery as a “safety net” for those 
students still not succeeding. Developed by the Reading Recovery program at Ohio State, 
the Literacy Collaborative (LC) provides support for students who are not receiving RR. 
LC involves the whole school — especially teachers and families — in a comprehensive 
and reflective approach to literacy instruction, which is appropriate for all children. This 
program is relatively new and lacks a confirmatory research base. 

Full-Day Kindergarten: Full-day kindergarten (FDK) is not funded by the state of 
Indiana as part of the state formula. Based on research-based models (e.g., Elicker & 
Mathur, 1997; Humphrey, 1988; Sheehan, Cryan, Wiechel, & Bandy, 1991), EIGP 
funded FDK programs that provided enhanced approaches to literacy and math. 

First Steps: A classroom-based language development model that serves as a 
teacher resource for closing the loop between diagnostic observation of child 
development and classroom instruction. First Steps provides teaching strategies, specific 
outcomes, and parent involvement ideas for each stage of child development. The 
research base is modest and mostly descriptive (Australian Council for Educational 
Research, 1993a, 1993b; Deschamp, 1995). 



Even Start : An early intervention program that aims to help break the poverty 
cycle by improving educational opportunities for low-income families. Even Start is not 
intended to affect early reading directly, but to create a developmentally appropriate 
home environment. A few federal studies have assessed the effects of the program, 
indicating it helps students make normal educational progress (e.g., Connor-Tadros, 

1996; Gamse, Conger, Elson, & McCarthy, 1997). 

Accelerated Schools Project : A schoolwide reform model, Accelerated Schools 
Project (ASP) is based on the notion that students in at-risk situations can learn at an 
accelerated pace by offering enriched curricula and instruction similar to that used for 
gifted education. ASP was designed as an inquiry-based professional development model 
with a clearly articulated philosophical base, which encourages active and reflective 
experimentation and evaluation. The program has a substantial research base (e.g., 
Finnan, St. John, Slovacek, & McCarthy, 1996), but the research that considers student 
outcomes is limited (e.g., Knight & Stallings, 1995; McCarthy & Still, 1993). 

Four Blocks: Multi-level, multi-method instruction provided in a framework that 
provides an organized, systematic structure for providing early literacy instruction. The 
four “blocks” of the program are: guided reading, self-selected reading, writing, and 
working with words. The model was proposed by Cunningham (1991) and has a modest 
research base (Cunningham, Hall, & Defee, 1991, 1998) conducted by the model 
advocates. 

Direct Phonics Instruction 

The third reform hypothesis is that direct instruction in letter/sound relationships 
(i.e., direct phonics instruction) can improve early reading acquisition. The most 
influential study on phonological awareness and the alphabetic principle is by Barbara 
Foorman and her colleagues (Foorman, Francis, Fletcher, Schatschneider, & Mehta, 
1998), a study that was independent of specific reform models. Recently, in response to 
critics, Foorman and her colleagues summarized their argument: 

However, we do maintain that there are some instruction principles that teachers and 
schools can use to enhance the reading achievement of at-risk children, and that it makes 
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sense to demonstrate effective implementation of these instructional principles before 
investing in more complex solutions. (Foorman, et al., 2000, p. 27) 

Thus, while Foorman and her colleagues do not overtly argue against taking 
comprehensive approaches to early reading improvement, they do argue for a specific 
method as a first priority. This argument was based on research that found: 

Controlling for differences in age, ethnicity, and verbal IQ, we found that children in the 
direct code (DC) approach improved in word reading at a faster rate and had higher word 
recognition skills in April than children receiving the implicit code (IC) approach (either 
research-based IC or district’s standard IC). More importantly, children in all 
instructional groups with higher phonological processing scores in the beginning of the 
year demonstrated improvement across the year. (Foorman, Fletcher, Francis, & 
Schatschneider, 2000, p. 29) 

Thus, this argument rests on research that shows that students who have instruction in 
direct coding learn to read words faster and that students who have skills in phonological 
processes learned to read faster. Some states have actually required phonics instruction in 
teacher education based on this line of research. The Snow report (Snow, et al., 1998) 
concludes that schools should first emphasize direct instructional approaches. However, it 
is not clear whether the particular reform models they advocated (e.g., Success for All) 
actually influenced schools to place more emphasis on direct instruction. Several states — 
including Texas, Washington, California, New York and Wisconsin — have required 
direct phonics instruction (Allington & Woodson-Jiron, 1998; Taylor, et al., 2000). This 
approach to policy takes the claim made by Foorman, et al. (1998) quite literally. That is, 
these researchers essentially claim that direct phonics instruction is a necessary first step 
and legislators in these states have taken action requiring this instructional approach. The 
implicit political claim made in this approach is that requiring all schools to use an 
explicit approach to phonics instruction will increase the number of students who learn to 
read and make normal educational progress. 

These arguments have had an influence on the evolution of the EIGP in Indiana in 
the sense that direct instruction is widely advocated. Many schools in Indiana used direct 
approaches in their early reading instruction. Further, the survey instrument developed 
for this study allowed us to measure the extent to which schools were using direct 
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instruction and related classroom practices. Thus it was possible to test the direct 
instruction hypothesis. 

Testing Hypotheses 

The EIGP was implemented in fall 1997 and has been funded for the past four 
years. The program was funded by the state at $4 million per year during the three years 
examined in this study. Three years of surveys (1997-98, 1998-99, and 1999-2000) were 
conducted of schools funded by EIGP along with a random sample of schools that did not 
receive funding. The surveys asked questions about actual classroom practices in both 
sets of schools and provided sufficient data to examine the effects of the three approaches 
to educational improvement on: 

• Two outcomes related to attainment/equity (rates of special education referral 
and retention in grades 1-3); and 

• One outcome related to achievement (the percentage of third grade students 
passing the state reading test, ISTEP+, in the fall term after the intervention). 

Specifically we examined the effects of school characteristics, funding, type of 
intervention, professional development features, parent involvement features, and 
classroom practices (factor scores) on these outcomes. A factor that combined practices 
related to direct instruction was included in the regression models. These analyses 
provided a relatively comprehensive assessment of the impact of these reform models, 
which are examples of comprehensive reforms. Further, since the survey instrument 
asked questions about professional development and direct instruction, it was possible to 
analyze information related to all three hypotheses. 

Research Approach 

In this study we use three years of survey data on schools in Indiana, collected as 
part of a comprehensive evaluation of the Early Intervention Program in Indiana. Below 
we describe the survey we used, the logical model used in the study, and the instruction 
and related factors developed from the study of the entire three-year database. 



Survey and Response 

The survey instrument, the Early Literacy Intervention Survey, 1 included 
questions about the school, types of interventions used, the features of the early reading 
program, the number of students referred and retained, and enrollment information that 
could be used to impute special education referral and retention rates. In addition, we had 
access to a state-level database with information on test scores. 

The survey assessed the frequency of use of nine organizational and structural 
features (ability grouping, basal readers, child-initiated learning centers, independent 
reading, one-on-one tutorials, pullout instruction, small groups, systematic evaluation, 
and trade books). It also assessed the frequency of use of ten classroom instructional 
methods (Big Books, cooperative learning, creative writing and/or essays, drama, 
emergent spelling, paired reading, phonics, reading aloud, reading drills, and 
worksheets/workbooks). For these frequency-of-use questions, survey participants were 
asked to respond on a five point scale for the extent of use from 1 for “never” to 5 for 
“every day” for both the current year and the prior year by grade level (K, 1, 2, 3). The 
survey also asked whether five types of professional development processes (certified 
training, certified specialist, in-service workshops, networking, and opportunity for 
collaboration) and five features related to parent involvement (book distribution, family 
literacy, paired reading, parent conference, and parent volunteers) were used in 
kindergarten through grade three. 

This study reports analyses for three years of surveys of funded and comparison 
schools. The overall response rate across the three years was 61%. Comparison schools 
were half as likely to be surveyed as funded schools. Therefore comparison schools were 
weighted by 2, to adjust for the probability of being surveyed. 

Statistical Methods 

The study used descriptive statistics, factor analysis, and multiple regression. The 
descriptive statistics describe the population characteristics. 



1 Copies of the survey can be obtained on request from the Indiana Education Policy Center. A simplified 
version of the instrument that can be used to survey teachers is available on line (St. John, & Bardzell, 
1999 ). 




? 



id 



A factor analysis was run for 19 variables related to instructional and classroom 
program features for the entire population. Specifically, the average Likert score for 
grades 1-3 was imputed for the 19 program features for instruction and 
structural/organization features of the survey instrument. A conservative factor-loading 
minimum of .50 was used. Missing items were replaced with mean values. 

Ordinary least squares (OLS) regression was used to examine the influence of 
predictor variables on the three outcomes. We present R , plus three levels of significance 
(.01, .05, and .10) for each predictor variable. Since .10 is only a moderately significant 
association, we make note of this moderate association in the text, so the reader will not 
place undue emphasis on this statistical relationship. 

Model Specifications 

A multiple regression model was developed to assess the effects of school 
characteristics, intervention type, professional development and instruction, and related 
factors on three outcomes: special education referral rates for grades K-3, retention rates 
for grades K-3, and passing rates of third grade ISTEP+ reading. The analyses for the 
first two outcomes (i.e., referral and retention rates) considered three years of surveys. 

The analysis for one outcome (i.e., test pass rates) considered two years of surveys. The 
blocks of variables, added sequentially, were: 

• School Characteristics : Whether a school was funded, the average ISTEP+ score, 
the percentage of students receiving free or reduced lunch, the percentage of 
minority students, and school locale (urban and rural schools were compared to 
students in other locale types). 

• Intervention Type : Reading Recovery 2 , Success for All, Literacy Collaborative, 
full-day kindergarten, First Steps, Even Start, Accelerated Schools 3 , and Four 
Blocks were coded as dichotomous variables. Some schools had more than one 
of these reform models. 



2 This variable coding included schools with Reading Recovery whether or not they were funded through 
EIGP. 

3 The EIGP did not fund any Accelerated Schools, but this intervention type was discussed in documents 
disseminated through the program (St. John & Bardzell, 1999) and there were a few Accelerated Schools in 
the state. 
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• Professional Development: Whether reading teachers are required to be certified, 
whether certified specialists are brought in for training sessions, whether in- 
service workshops were used, whether teachers networked with teachers in other 
schools, and whether teachers collaborated within the school on reading 
instruction are included in professional development. 

• Parent Involvement: Book distribution, family literacy, paired reading (parent to 
child), parent conferences, and parent volunteers were included as dichotomous 
variables. 

• Instruction and Related Factors: We included the scores for the nine 
structural/organizational factors and the ten classroom instructional methods. 

Instructional and Related Factors 

Because of the large number of program features related to instruction and the 
organization of reading programs at the grade level, we decided to conduct a factor 
analysis of the instructional and structural/organizational features. (The factor analysis is 
presented in St. John, et al., 2000). 

• The Connected-Text Approaches factor includes independent reading, 
cooperative learning, creative writing, emergent spelling, paired reading (student- 
to-student), and reading aloud. Schools that make use of these methods combine 
techniques that engage students in the learning process. 

• The Direct/Explicit Approaches factor combines basal readers, phonics 
instruction, reading drills, and worksheets/workbooks. Schools that use 
Direct/Explicit Approaches emphasize systematic approaches to teaching the 
components of language and reading. 

• The Child-Centered/Expressive Approaches factor includes child-initiated 
learning centers, Big Books, cooperative learning, and drama. These instructional 
approaches place an emphasis on the development of the whole child and peer 
engagement among children. 

• The Small Group/T utorial Approaches factor combines ability grouping, one-on- 
one tutoring, pullout instruction, and small groups. Schools that use these 
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techniques place more emphasis on classifying children and accelerating the 
learning of some, while addressing developmental needs of others. 

• The Trade-Books Approaches factor combines trade books and Big Books, but 
de-emphasizes basal readers. In this approach, schools use texts that are 
literature-based and engaging for students, rather than structured elements of 
reading programs that emphasize increasing levels of difficulty. 

Of these factors, the variables included in the Direct/Explicit Approaches factor 
are closely aligned with the direct approaches advocated by Foorman and colleagues 
(Foorman, et al., 1998; Foorman, et al., 2000). The other factors would seem more 
closely aligned with the more comprehensive approaches advocated by Taylor and other 
reform advocates. 

Limitations 

This study has a few limitations that merit consideration by readers. First, our 
analyses consider school-related outcomes rather than individual outcomes. While most 
reading research focuses on individual students, we felt it was important for the funding 
agency to understand whether their funding influenced student outcomes at the school 
level. This approach is consistent with the ways school outcomes are frequently reported 
to legislators and the public. Thus, this approach was appropriate for a policy study of 
this type. 

Second, the survey asked respondents to answer questions about program features 
for all grade levels, rather than asking each teacher to respond to a questionnaire. We 
considered this approach appropriate for this initial test of the study methodology. In the 
future we plan to extend the method to include a survey of teachers, which would mean 
we could examine both school level and classroom level outcomes. 

Third, we assumed the schools that responded to the survey were representative of 
all of the schools in the funded and comparison groups. This assumption was necessary 
because of the statistical methods, but it seems a reasonable assumption for this study 
population. While funded schools and comparison schools had different response rates, 
these differences seem reasonable given that funded schools had more motivation to 
respond as a result of their involvement in the program. 



Fourth, while many of the comparison schools in Indiana used direct instruction 
methods, the funded interventions did not place a substantial emphasis on these practices, 
an issue considered below. In the summer of 2000, the IDOE provided statewide training, 
developed a web page, and developed web-based courses (for continuing education 
courses) that emphasized direct instruction. In the next study it may be possible to further 
test this hypothesis. 



Findings 

Sample Characteristics 

The characteristics of the Indiana schools included in this study are presented in 
Table 1. The three outcomes provide very different measures of school success. The 
average rates for special-education referral (5%) and grade-level retention (2%) were 
relatively low. In contrast, an average of 69% passed the ISTEP+ reading/language arts 
test. 

The schools in this study represent a great diversity in school characteristics. The 
responding schools had an average of 26% of free and reduced lunch and an average of 
13% minority students. About two-tenths were from city locales (18.5%) and about one- 
third rural (31.9%). The other half were located in suburban and town locales. In 
addition, about one-third of the sample received one year of funding (32%) and 1 1.3% 
received two years of funding, with very few schools receiving funding all three years. 

Three types of interventions represented the majority of comprehensive 
interventions: Reading Recovery (40.8%), full-day kindergarten (12.3%), and Four 
Blocks (12.5%). These methods were being encouraged by the IDOE through different 
initiatives during the period of the study. Local political forces influenced the number of 
schools that applied for state funds to implement these methods. 4 Much smaller 
percentages of the schools used Success for All, Literacy Collaborative, and other 
methods. However, this list of reforms does not correspond perfectly with receipt of 
funds. For example, not all Accelerated Schools received funding through the EIGP. 

Most schools in the sample used in-service workshops (77%), networking 
(66.1%), and collaboration (73.5%). Certified training and certified specialists, 

4 These approaches were formally encouraged by the Indiana Department of Education. 
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professional development methods linked to Reading Recovery in Indiana, were used in 
about one third of the schools in the sample. 

Parent involvement was widely used as a strategy in early reading programs. 
About half of the schools used book distribution. Most used parent conferences, parent 
volunteers, and paired (parent-to-child) reading. About one-third had family literacy 
programs. 

Referral Rates 

The analysis of referral rates, represented as a sequential set of regression models 
(Table 2), reveals that all but one of the variables that were significantly associated with 
special education referral had a consistent effect across all of the models. Each set of 
variables is considered below. 

First, two variables related to school characteristics had a significant association 
with special education referrals across all of the models. Having high average ISTEP+ 
reading scores the prior year was negatively associated with referrals, indicating that 
schools with higher test scores had lower referral rates, controlling for other school 
characteristics. Further, the percentage of students on free and reduced lunch was 
positively associated with special education referrals, indicating a relationship between 
poverty and special learning needs. None of the other school characteristics was 
significantly associated with referral. 

Second, four of the intervention types had consistently significant associations 
with referral rates. Schools with Success for All, Literacy Collaborative, and First Steps 
were negatively associated with special education referral rates. Controlling for other 
variables in the model, these interventions help more students stay in the educational 
mainstream. In contrast, Four Blocks was significant and positively associated with 
referral rates. 

Third, none of the professional development variables were significant in any of 
the analyses of special education referrals. This indicates that professional development 
for reading was not associated with referral rates. 

Fourth, one of the parent involvement variables, family literacy, was modestly 
significant (.10 alpha) in the final step, but was not statistically significant when the 



instructional factors were taken into account. Controlling for all variables in the model, 
family literacy was associated with lower referral rates. This indicates that family literacy 
programs help families and schools to work together in a complementary way to keep 
more children in school. 

Fifth, two of the factors related to classroom practices were significant. 
Connected-Text Approaches and Ability Grouping/Pullout Approaches were significant 
and positively associated with special education referral. Schools that emphasized these 
practices had higher referral rates, controlling for other variables in the model. 

Retention Rates 

The analyses of retention rates reveal a complex pattern of relationships between 
independent variables, as well as a number of direct relationships between predictor 
variables and retention rates (Table 3). Each set of variables is examined below. 

First, five variables in the first model were significant in at least one version of 
the model. Two of the variables — the percentages of students getting free lunch and of 
minority students — were significant and positively associated with retention across all of 
the models. Schools with more low-income and minority students had more retention. 

Two of the funding variables — receiving one year of funding and receiving two 
years of funding — were significant and negatively associated with retention in the first 
two steps of the analyses. Thus, controlling for other school characteristics, receiving one 
or two years of funding through EIGP was associated with keeping more students on 
grade level. Both variables were highly significant (.05) in the first step, modestly 
significant in the second step (.10) and not significant in any of the subsequent steps. The 
drop in the extent of significance across both steps indicates that the influence of funding 
is related to the variables in the each of these steps. The significance of the two funding 
variables was mitigated when the types of interventions were added in the second model. 
This is a logical finding given that the funding supported the programs that many schools 
implemented. Another portion of the significance of the funding variables was related to 
the professional development opportunities in the school. Specifically, when having an 
opportunity to collaborate was considered, the funding variables were no longer 
significant. This suggests that receiving EIGP funds enabled teachers to collaborate. 
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another desirable outcome of funding, at least according to the professional development 
hypothesis. 

One variable, being in a rural locale, was modestly significant (.10) after the 
program types were added to the model. This suggests a confounding relationship 
between intervention types and the rural locale. Further analysis of the types of programs 
implemented in rural locales would be needed to discern why there was such a 
confounding relationship. 

Second, two variables related to intervention types were significant in at least one 
step of the model. The Literacy Collaborative was modestly significant (.10) and 
negatively associated with retention in the first two versions of the analysis. There is a 
confounding relationship between the Literacy Collaborative and variables related to 
parent involvement. Since the Literacy Collaborative encourages family involvement in 
reading, this was not an unexpected finding. 

Reading Recovery was also modestly significant (.10) after professional 
development variables were added to the model (steps 3, 4, and 5). Because Reading 
Recovery provides certified training and because schools with this program had certified 
specialists in the first grade, we expect the confounding relationship was attributable to 
these variables. Certified training and certified specialists were positive but not 
significant when they entered the model. The positive association in these variables offset 
the negative association of Reading Recovery in the last three steps. 

Third, only one professional development variable was significant across all of 
the models. Having the opportunity to collaborate was negatively associated with 
retention, indicating that when teachers have the opportunity to collaborate it is easier for 
them to communicate about the types of educational programs that will enable more 
children to achieve on grade level. 

Fourth, two variables related to parent involvement were significant. Paired 
reading between parents and children was associated with lower retention rates, while 
parent conferences were associated with higher retention rates. Paired reading is directly 
related to literacy instruction and is an integral feature of Literacy Collaborative, which 
could explain the association between this program type and professional development. 
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Finally, four of the factors were significant. Connected-Text Approaches and 
Trade Books were significant and negatively associated with retention rates, indicating 
these patterns of practice help more children perform on grade level. Direct/Explicit 
Approaches and Child-Centered/Expressive Approaches were significant and positively 
associated with retention rates, indicating these methods were associated with having 
fewer students perform on grade level. 

Test Pass Rates 

The analyses of the influence of predictor variables on pass rates for standardized 
tests (Table 4) also indicated a complex pattern of relationships between predictor 
variables. Three of the school' characteristic variables had a consistent significant 
relationship with the outcome across each version of the model. Schools with higher 
average ISTEP+ scores during the base year had higher pass rates the next year. In 
contrast, the percentages of minority and low-income students were negatively associated 
with pass rates across all models. 

In this analysis, having one year of funding was significant and negatively 
associated with pass rates in the first two models. This variable was only modestly 
associated with pass rates in the third model (.10), and was not significantly associated 
with the outcome in the last two models. Thus funding had a confounding relationship 
with both professional development (and especially the opportunity to collaborate) and 
parent involvement. This reveals the opposite pattern as the prior analysis. Whereas the 
combination of funding and the opportunity to collaborate seemed to help keep more 
children in the classroom, the similar combination of variables was associated with lower 
pass rates. 

Three of the intervention types had significant associations with test pass rates. 
Success for All had a strong negative association with pass rates across all of the models. 
In contrast, both Literacy Collaborative and ASP had modest (.10) positive associations 
with higher pass rates after parent involvement was considered. Literacy Collaborative 
and Accelerated Schools Project place a substantial emphasis on parent involvement. 

Opportunity to collaborate was significant and negatively associated with pass 
rates. In combination with the analysis of the retention rates above, these findings suggest 



a complex pattern of confounding relationships between funding, professional 
development, and student outcome. To better understand these relationships, we reflect 
below on this evidence in relation to the reform hypotheses. 

None of the factors related to classroom practices was significant. Prior analyses 
that had considered the type of program and type of program funding separately, revealed 
a positive relationship between Direct/Explicit Approaches and pass rates (St. John, 
Manset, Chung, Simmons, & Musoba, 2000; St. John, Manset, Chung, Simmons, 
Musoba, Manoil, & Worthington, in press). The present model provides a better way to 
control for the effects of funding and program type. Therefore, controlling for funding 
and program type, we conclude that none of the patterns of classroom practice were 
associated with pass rates independent of the effects of the types of programs 
implemented and the receipt of funding. 

Evidence Related to Reform Hypotheses 

Three rationales influenced the development and evolution of the Early 
Intervention Grant Program in Indiana. By considering the results above in relation to the 
three hypotheses, it is possible to untangle further how government-sponsored 
interventions influence learning outcomes. 

Professional Development 

The professional development hypothesis was that: teachers need time for 
professional development that includes collaboration strategies for improving 
educational outcomes. These analyses reveal a pattern of relationship between funding 
and the opportunity to collaborate. In two of the three analyses, the effects of receiving 
funding disappeared when the effects of the opportunity to collaborate were also 
considered. There are three distinct aspects of this pattern of findings that merit attention. 

First, this pattern of findings seems consonant with the model of professional 
development (Bull & Buechler, 1996) that the state of Indiana has encouraged schools to 
emphasize in all categorical grant applications (St. John, Ward, & Laine, 1999). This 
finding indicates that the state’s encouragement of professional development had an 
influence on the strategies schools used to involve teachers in their reform efforts. 
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Second, these two findings — that funding and the opportunity to collaborate were 
associated with less retention and higher test pass rates — indicate a relationship between 
keeping children in the mainstream and passing tests. Indeed, one way to improve pass 
rates is to increase retention. When teachers have time to collaborate, they exhibit an 
increased capacity to keep more students achieving at grade level. However, there is an 
implied tradeoff between two sorts of outcomes — keeping children on grade level and 
having higher pass rates. 

Third, two years of funding with a comprehensive reform method overcame the 
implicit tradeoff. Having one year of funding and having two years of funding were 
associated with lower retention rates, while only having one year of funding was 
associated with having lower test pass rates. Two years of funding provided more time 
for teachers to work through the problems associated with creating learning environments 
that help more children to achieve on grade level. If schools had two years of funding 
before children entered the third grade, children benefited from the programs. Having 
supplemental programs in both first and second grades apparently made a difference for 
students. Since children took the ISTEP+ test at the start of third grade, the two years of 
funding would have been sufficient to affect the reading programs prior to taking these 
achievement tests. 

Not only do these three interpretations seem to support the professional 
development hypothesis, but they also suggest the possibility of a deeper pattern. One 
line of argument about school reform is that restructuring processes that involve teachers 
and parents can create caring learning communities that engage more children in active 
learning (St. John, Griffith, & Allen-Haynes, 1997). Not only do the findings above 
indicate a relationship between funding and opportunity to collaborate among teachers, 
but parent involvement also seems to be an important relationship with respect to test 
passing rates (see Table 4). “People” variables have the most impact, indicating more 
attention should be given to creating caring communities that support the development of 
children in schools. This pattern of relationships among variables merits further 
consideration in future studies. 
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Comprehensive Reform 

A second hypothesis was that: comprehensive and cohesive reform approaches 
are essential to improving schools ’ outcomes. This hypothesis is not only a long-standing 
assumption of reforms such as Reading Recovery (Clay, 1993), but it is also a counter- 
argument to newer claims about direct phonics instruction. The findings of this study 
reinforce the notion that comprehensive and cohesive approaches to reform can influence 
educational outcomes, but it also reveals that the effects of reform models are far from 
monolithic. 

The effects of the distinctive reform models are summarized in Table 5, along 
with an interpretive comment related to these reforms. These analyses not only controlled 
for the effects of funding and school characteristics, but also considered confounding 
relationships with professional development, parent involvement, and patterns of 
classroom practice. Different reform models had different effects, reinforcing the notion 
that not only do different comprehensive reform models have different features but they 
also have different effects. 

It is abundantly apparent that the reforms had different effects. One of the reform 
models. Literacy Collaborative, had the desired association with all three outcomes, 
indicating a comprehensive impact of student outcomes. Another model, Success for All, 
had a desired association with one outcome (special education referral) and an undesired 
association with another (achievement tests), suggesting a tradeoff implied in selecting 
this model. A third model. Four Blocks, had only one undesired association (related to 
higher special education referral). The other models noted in the table (Reading 
Recovery, full-day kindergarten, and Accelerated Schools) had one desired outcome. 

It was also apparent that the effects seemed directly related to the design features 
in these models. For example, the finding that Reading Recovery was associated with 
higher retention rates seems directly related to the program’s focus on raising the reading 
level for first grade children who are having trouble learning to read. Even the lower- 
than-expected findings associated with Four Blocks seem related to the lack of emphasis 
on professional development and parent involvement in this model. Professional 
development was integral to the state reform strategy and it had a direct effect on some 
outcomes, but Four Blocks per se was not positively associated with these outcomes. 
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Clearly schools should consider the design features of possible reform models when they 
are considering a reform strategy. 

Direct Instruction 

The third hypothesis was that: direct instruction in letter/sound relationships (i.e., 
direct phonics instruction) can improve early reading acquisition. The findings presented 
here do not support the direct instruction hypotheses. Earlier analyses, earlier studies that 
did not control for the years of funding, found that Direct/Explicit Approaches were 
associated with higher grade level retention rates and with higher passing rates on 
standardized tests (Manset, St. John, Hu, & Gordon, in press; St. John, Manset, Chung, 
Simmons, & Musoba, 2000). These earlier studies did not examine the effects of funding 
and the types of reforms adopted, but rather used treatment variables that combined 
funding and reform type. Thus, the current study has better logical and statistical controls 
for the effects of funding than the earlier studies, which probably explains the difference 
in findings (given that the funding variable has the reverse effect noted in these earlier 
studies). Thus we conclude that the earlier finding — that Direct/Explicit Approaches was 
associated with higher pass rates on standardized tests and lower retention — was an 
artifact attributable to the first year effects of funding (i.e., the first year of new 
practices). However, regardless of which set of findings one accepts, there is not strong 
support for the direct instruction hypothesis. 

The partial correlations between the program types of the factors provide more 
explanations for these findings (see Table 6). These analyses reveal that most of the 
comprehensive reform models we examined were correlated with factors other than 
Direct/Explicit Approaches. These models emphasize literature-rich instruction and de- 
emphasize direct phonics instruction, worksheets, and basal readers. None of the reform 
models was significant and positively correlated with the Direct/Explicit Approaches 
factor. In fact, schools with four of the reform models — Reading Recovery, Success for 
All, Literacy Collaborative, and Four Blocks — were actually negatively associated with 
the Direct/Explicit Approaches factor. 

This supports the notion that a balanced approach to reading instruction may be 
needed. Apparently there really is a difference between the reforms, such as Success for 
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All, that were endorsed by the National Research Council (Snow, Bums, & Griffith, 

1998) and direct approaches to reading intervention that are now so widely advocated. 
Success for All had a negative correlation with the Direct/Explicit Approaches factor, 
which includes more emphasis on phonics and related practices. Similarly, several of the 
other reform models had similar patterns of practice as Success for All. Indeed, schools 
with Success for All — along with schools with Reading Recovery, Literacy Collaborative 
and other locally developed programs — were associated with Connected-Text 
Approaches and Ability-Group/Pullout Approaches (Table 6). Thus, a comprehensive 
approach may provide a literature-rich environment and supplemental support for 
children who are having trouble learning to read. 

In the summer of 2000, the IDOE sponsored statewide training on a “phonics 
toolkit” and development of web-based continuing education in methods for promoting 
phonemic awareness as a part of classroom instruction. Perhaps these workshops will 
introduce new methods in direct instruction that will have the desired influence on 
student outcomes. This development certainly merits further study. 

Thus, it may be too early to assess adequately whether the direct instruction 
hypothesis holds up to empirical evidence. In Indiana, at least, research to date does not 
support the direct-instruction hypothesis. Therefore, we think further analyses are needed 
to test more fully whether these reform claims hold up when reforms are implemented 
that emphasize phonics and other direct approaches to early reading instruction. 

Conclusions and Implications 

The primary conclusion from this study is that the implicit hypotheses — or reform 
beliefs — advocates hold often oversimplify the complexity of reforming reading 
programs in ways that improve student outcomes. While the idea that a research base can 
inform policy is compelling, adhering to a political claim based on research that is 
conducted by reform advocates may be shortsighted. Policymakers and educators should 
be more skeptical about the claims made by researcher/reformers, including those who 
extrapolate about the value of direct phonics instruction from related research. While 
there is some compelling evidence to support the core aspects of the claims examined 
here, each set of claims was more complex than portrayed by the reform advocates. Thus, 
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while the new wave of research-based reading reforms holds promise, it is far from a 
panacea for educators. 

This study indicates it is important to consider the role of the various rationales 
(or policy arguments) that influence the evolution of a grant program when evaluating the 
effects of the program. In particular, we found that the state’s emphasis on embedding 
professional development in all grant proposals explained some of the direct effects of 
funding, after the types of interventions were controlled. This finding reveals that placing 
an explicit emphasis on professional development in the funding process can have a 
positive impact on students if the intervention is sustained for at least two years. While 
specific comprehensive reform models also had effects, different models had different 
effects. This means that schools should choose program types that meet their needs; that 
is, address deficiencies in their existing programs. Thus, the general rationale that 
‘comprehensive reforms improve student outcomes seems to overgeneralize a complex set 
of relationships between reforms and student outcomes. Further, there is reason to raise 
questions about the practical meaning of the direct instruction rationale. Specifically, it is 
difficult to discern how these practices actually influence student outcomes in schools, 
given that many schools already use these practices. 

These findings have three important implications. First, it is crucial that 
policymakers include comprehensive evaluation studies when they undertake major new 
reforms. Too often state education reforms do not include evaluations. Nor do evaluations 
routinely test the assumptions made by reform advocates. Policymakers need to ponder 
the possibility of the null hypothesis. Indeed, the results of this study suggest that reform 
strategies that allow educators to select intervention designs that meet their educational 
needs may have a greater impact than mandating single reform models. For example, the 
national implementation of the Comprehensive School Reform Demonstration has 
enabled many schools to choose from among possible reform models. This approach, 
which is similar to the approach used by the state program examined in this report, seems 
a more reasonable approach than advocating a specific reform method in the legislation 
for education reform. 

Second, it is crucial that evaluators think critically about the claims made by 
various educational reformers. These claims should be evaluated and openly discussed. 
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The rationales used by reformers are generally based on research that tests a single claim. 
Since the funding programs respond to a range of policy arguments in their design and 
evolution, it is important that researchers examine the impact of the specific activities that 
are required by a funding agency in order to secure funds. It also is necessary to control 
for the contexts in which reforms are implemented, as well as for classroom practices in 
comparison schools. In this study, we were careful to discern the features of various 
reform models and to ask educators about whether those features were frequently used in 
their schools. This enabled us to assess the impact of patterns of classroom practices on 
educational outcomes and to control for these patterns when we assessed the impact of 
reform models. When we took this step, it became apparent that different reform models 
had different effects. Further, we found that the designs themselves helped to explain the 
impact of the reform models. 

Third, it is important that educators take the opportunity to think critically about 
the features of various reform models and the ways these features link to educational 
outcomes and the ways the reform models would change their education practices. To do 
this, they need to be able to assess their current practices, to consider the features of 
current reading programs, and to compare these practices to the features of various 
reforms. Some work has been done in producing guides that can help educators assess 
current practices (e.g., North Central Regional Education Laboratory, 2000; St. John & 
Bardzell, 1999), but more work is needed. Given that the new wave of research-based 
reading reforms is creating opportunities for educators to make informed choices about 
improvement strategies, a better effort should be made to disseminate useful information 
into the hands of educators who are making choices about reform strategies. 
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